---
id: frame-processors
title: Frame Processors
sidebar_label: Frame Processors
---

import Tabs from '@theme/Tabs'
import TabItem from '@theme/TabItem'
import useBaseUrl from '@docusaurus/useBaseUrl'

<div class="image-container">
  <svg xmlns="http://www.w3.org/2000/svg" width="283" height="535">
    <image href={useBaseUrl("img/frame-processors.gif")} x="18" y="33" width="247" height="469" />
    <image href={useBaseUrl("img/frame.png")} width="283" height="535" />
  </svg>
</div>

## What are frame processors?

Frame Processors are JavaScript functions that are called for each Frame the Camera "sees".

Inside those functions you can analyze the Frame in realtime using native [Frame Processor Plugins](#native-frame-processor-plugins-community) or draw directly onto the Frame using [Skia Frame Processors](skia-frame-processors).

For example, the [vision-camera-image-labeler](https://github.com/mrousavy/vision-camera-image-labeler) plugin can detect objects at 60+ FPS:

```jsx
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const objects = detectObjects(frame)
  const label = objects[0].name
  console.log(`You're looking at a ${label}.`)
}, [])

return <Camera frameProcessor={frameProcessor} />
```

Due to their extensible plugin-based nature, Frame Processors can be used for all sorts of things, like:

* **ML** for **face recognition**
* Using **Tensorflow**/**TFLite**, **MLKit Vision**, **Apple Vision** or other libraries
* Creating **realtime video-chats** using **WebRTC** to directly send the camera frames over the network
* Creating scanners for custom codes such as **Snapchat's SnapCodes** or **Apple's AppClips**
* Creating **snapchat-like filters**, e.g. draw a dog-mask filter over the user's face
* Creating **color filters** with depth-detection
* **Drawing** boxes, text, overlays, or colors on the screen in realtime
* Rendering **filters** and shaders such as Blur, inverted colors, beauty filter, or more on the screen

## Installation

Frame Processors require [react-native-worklets-core](https://github.com/margelo/react-native-worklets-core) 1.0.0 or higher. Install it:

```sh
npm i react-native-worklets-core
```

And add the plugin to your `babel.config.js`:

```js
module.exports = {
  plugins: [
    ['react-native-worklets-core/plugin'],
  ],
}
```

## The `Frame`

A Camera Frame is a GPU-based pixel buffer, usually in YUV or RGB format.
The [`Frame`](/docs/api/interfaces/Frame) object contains these GPU-based pixel buffers, and exposes them to JavaScript. For example, to log information about the Frame such as it's resolution or pixel format, simply access it's properties:

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log(`Frame: ${frame.width}x${frame.height} (${frame.pixelFormat})`)
}, [])
```

To directly access the Frame's pixel data use [`toArrayBuffer()`](/docs/api/interfaces/Frame#toarraybuffer):

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  if (frame.pixelFormat === 'rgb') {
    const buffer = frame.toArrayBuffer()
    const data = new Uint8Array(buffer)
    console.log(`Pixel at 0,0: RGB(${data[0]}, ${data[1]}, ${data[2]})`)
  }
}, [])
```

While you can process the Frame's pixel data in JavaScript, it is recommended to use native [Frame Processor Plugins](#native-frame-processor-plugins-community) instead for better performance and GPU-acceleration.

:::tip
At 4k resolution, a raw Frame is roughly **12MB** in size, so if your Camera is running at 60 FPS, roughly **700MB** are flowing through your Frame Processor **per second**.

Such amounts of data cannot be copied or serialized fast enough, so VisionCamera uses [JSI](https://reactnative.dev/docs/the-new-architecture/landing-page#fast-javascriptnative-interfacing) to directly expose the GPU-based buffers from C++ to JavaScript.
:::

## Native Frame Processor Plugins

Usually JavaScript is not fast enough to process heavy amounts of data in realtime as it does not have access to GPU-acceleration directly.
Instead, you should use native languages (Objective-C/Swift, Java/Kotlin or C++) to write the actual heavy processing code, such as face detection or object detection.

VisionCamera provides a plugin system to directly and synchronously call into native languages, run any processing on the Frame, and return back to JavaScript with almost no noticeable overhead at all.

This plugin-based approach ensures maximum flexibility while still guaranteeing maximum performance.
For example, the `sensitivity` value can be tweaked on-the-fly via fast-refresh in development, or shipped with over-the-air updates in production - without a native rebuild needed.

```ts
const sensitivity = 0.8
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const faces = detectFaces(frame, { sensitivity: sensitivity })
  // ...
}, [sensitivity])
```

### Using Community Plugins

Community Frame Processor Plugins are distributed through npm. To install the [vision-camera-image-labeler](https://github.com/mrousavy/vision-camera-image-labeler) plugin, run:

```bash
npm i vision-camera-image-labeler
cd ios && pod install
```

That's it! 🎉 Now you can use it:

```ts
const { labelImage } = useImageLabeler()

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const labels = labelImage(frame)
  const label = labels[0].name
  console.log(`You're looking at a ${label}.`)
}, [labelImage])
```

Check out [Frame Processor community plugins](/docs/guides/frame-processor-plugins-community) to discover available community plugins.

### Creating native Frame Processor Plugins

VisionCamera provides an easy-to-use API for creating native Frame Processor Plugins, which you can use to either wrap existing algorithms (e.g. the iOS/Android ["MLKit Face Detection"](https://developers.google.com/ml-kit/vision/face-detection) API), or to build your own custom algorithms.

A Frame Processor Plugin is a single native class which contains an initializer and a callback function that receives the native `Frame` (`CMSampleBuffer` or `Image`):

<Tabs
  groupId="platform"
  defaultValue="ios"
  values={[
    {label: 'iOS Example (Swift)', value: 'ios'},
    {label: 'Android Example (Java)', value: 'android'}
  ]}>
<TabItem value="ios">

```swift title="FaceDetectorPlugin.swift"
@objc(FaceDetector)
public class FaceDetectorPlugin: FrameProcessorPlugin {
  private let sensitivity: Double

  public override init(proxy: VisionCameraProxyHolder, options: [AnyHashable: Any]) {
    super.init(proxy: proxy, options: options)
    sensitivity = options["sensitivity"] as Double
  }

  public override func callback(_ frame: Frame, withArguments args: [AnyHashable: Any]) -> Any {
    let imageBuffer = CMSampleBufferGetImageBuffer(frame.buffer)
    let faces = MLKit.detectFaces(imageBuffer, sensitivity)
    return faces.map { face in face.toJson() }
  }
}
```

</TabItem>

<TabItem value="android">

```java title="FaceDetectorPlugin.java"
public class FaceDetectorPlugin extends FrameProcessorPlugin {
  private final Double sensitivity;

  public FaceDetectorPlugin(VisionCameraProxyHolder proxy, Map<String, Object> options) {
    super();
    sensitivity = options.get("sensitivity") as Double;
  }

  @Override
  public Object callback(Frame frame, Map<String, Object> params) {
    Image image = frame.getImage();
    Array<Face> faces = MLKit.detectFaces(image, sensitivity);
    return faces.stream().map(f -> f.toJson());
  }
}
```

</TabItem>
</Tabs>

The native plugin can accept parameters (e.g. for configuration) and return any kind of values for result, which are automatically converted to- and from JavaScript values.

**🚀 See: ["Creating Frame Processor Plugins"](/docs/guides/frame-processors-plugins-overview) for more information.**

## Performance

The Frame Processor runtime is built with C++ and bridged to JavaScript with low-overhead using [JSI](https://reactnative.dev/docs/the-new-architecture/landing-page#fast-javascriptnative-interfacing).

### Considerations

As Frame Processors run synchronously in the video pipeline, you need to complete your processing before the next Frame arrives. At 30 FPS you have 33ms, at 60 FPS you only have 16ms per frame.

:::tip
The lower the video resolution of your [format](/docs/guides/formats), the faster your Frame Processor can execute.
For example, if your ML model is trained on 1280x720, use the format closest to that resolution:
```ts
const format = useCameraFormat(device, [
  { videoResolution: { width: 1280, height: 720 } }
])
```
:::

### Benchmarks

Frame Processors are _really_ fast. I have used [MLKit Vision Image Labeling](https://firebase.google.com/docs/ml-kit/ios/label-images) to label 4k Camera frames in realtime, and measured the following results:

* Fully natively (written in pure Objective-C, no React interaction at all), I have measured an average of **68ms** per call.
* As a Frame Processor Plugin (written in Objective-C, called through a JS Frame Processor function), I have measured an average of **69ms** per call.

This means that the Frame Processor API only takes ~1ms longer than a fully native implementation, making it **the fastest and easiest way to run any sort of Frame Processing in React Native**.

## Disabling Frame Processors

The Frame Processor API spawns a secondary JavaScript Runtime which consumes a small amount of extra CPU and RAM. Additionally, compile time increases since Frame Processors are written in native C++. If you're not using Frame Processors at all, you can disable them:

<Tabs
  groupId="environment"
  defaultValue="rn"
  values={[
    {label: 'React Native', value: 'rn'},
    {label: 'Expo', value: 'expo'}
  ]}>
<TabItem value="rn">

### Android

Inside your `gradle.properties` file, add the `enableFrameProcessors` flag and set it to `false`:

```groovy
VisionCamera_enableFrameProcessors=false
```

Then, clean and rebuild your project.

### iOS

Inside your `Podfile`, add the `VCEnableFrameProcessors` flag and set it to `false`:

```ruby
$VCEnableFrameProcessors = false
```

</TabItem>

<TabItem value="expo">

Inside your Expo config (`app.json`, `app.config.json` or `app.config.js`), add the `enableFrameProcessors` flag to the `react-native-vision-camera` plugin and set it to `false`:

```json
{
  "name": "my app",
  "plugins": [
    [
      "react-native-vision-camera",
      {
        // ...
        "enableFrameProcessors": false
      }
    ]
  ]
}
```

</TabItem>
</Tabs>

:::info
When [react-native-worklets-core](https://github.com/margelo/react-native-worklets-core) is not installed, Frame Processors are automatically disabled.
:::

<br />

#### 🚀 Next section: [Zooming](/docs/guides/zooming) (or [creating a Frame Processor Plugin](/docs/guides/frame-processors-plugins-overview))
